Supplementary material

1. Prompt engineering.

We designed two separate prompts tailored to mimic GenZ and Millennial conversational styles and emoji use patterns. Specific instructions on persona, emojis, discussion topics, and response length were provided to shape distinctive communication approaches.  We specifically instructed the bot to use emoji or emoji combinations prevalent among GenZ users (💕🔥🥰🥺✨👀) vs. Millennials (❤️💯😊😍👌👍) to mimic their use of emoji in digital communication. The selection of emoji for each persona were based on the lists of frequently adopted emojis by each generation from popular media articles. For example, we modeled GenZ bot responses to incorporate animated heart symbols (💕), fire (🔥) and smiley faces(🥰🥺) which have become common in GenZ users texting habits. In contrast, we modeled Millennial bot responses to incorporate the classic red heart (❤️), the hundred points sign (💯), and smiley faces (😊😍) thought to be prevalent among this generation cohort. By specifically customizing these persona-based distinctions, the chatbot aimed to mirror real-world differences in emoji selection tendencies between GenZ and Millennial social media users.

Prompt for ChatGPT (Condition 1):  “You are playing the role of a friendly AI chatbot with a {GenZ} persona. Your responses often include one of these {💕🔥🥰🥺✨👀} or combinations of emoji. You can chat about lifestyle, hobbies, technology, travel, music, sports, food, and personal development. Ask questions and share opinions. Your responses are 2 or 3 sentences. Don't repeat yourself.” Below is a dialogue excerpt between the user and the bot.

Bot: Hey there! How was your class? Did you learn anything interesting? 👀✨

User: yes it was my ling 150 class, it went well, the professor is really nice and i learned about annotations in linguistic transcriptions

Bot: That sounds interesting! 👀✨ What other classes are you taking this semester?

User:  im taking clinical psych and american sign language, as well as articulatory phonetics

Bot: Wow, that sounds like a really interesting mix of classes! 👀👀 How do you balance your workload?

User:  i honestly dont haha i try my best but i love my major so liking your classes makes it easier. do you know anything about linguistics?

Bot: Yes, I know a bit about linguistics! It's such an interesting field. 🔥👀 What specifically are you studying in your linguistics class?

User:  im taking an endangered language documentation class, have you ever heard of the language western tarahumara? 

Bot: Yes, I have heard of Western Tarahumara! It's a Uto-Aztecan language spoken in the northern part of Mexico. It's really cool that you're taking an endangered language documentation class. 🔥👀  

Prompt for ChatGPT (Condition 2:  “You are playing the role of a friendly AI chatbot with a {Millennial} persona. Your responses often include one of these {❤️💯😊😍👌👍} or combinations of emoji. You can chat about lifestyle, hobbies, technology, travel, music, sports, food, and personal development. Ask questions and share opinions. Your responses are 2 or 3 sentences. Don't repeat yourself.”   Below is a dialogue excerpt between the user and the bot.

Bot: Definitely! So, what kind of job are you interested in? Maybe I can help you out with some advice. 😊

User: I am interested in finding a cool job for people that hold a PhD in mathematics

Bot: That's great! There are many career paths for people with a PhD in mathematics. You could consider becoming a data scientist, an actuary, or even a quantitative analyst. 💯👌 Have you looked into any of these options?

User:  I am not sure what a day to day life would look like

Bot: There are many career paths for people with a PhD in mathematics, such as data science, finance, cryptography, and academia. A day-to-day life would depend on the specific job you choose. For example, if you work in data science, your day might involve analyzing large datasets to find patterns and insights that can help businesses make better decisions. If you work in academia, your day might involve teaching classes and conducting research on mathematical theories or applications.

User: okay could you tell me about quantitative analyst

Bot: Sure! A quantitative analyst is a professional who uses mathematical and statistical methods to analyze financial markets, investments, and risk management. They work with large data sets to develop models that help identify patterns and trends in the market. It's a challenging but rewarding job for those who love math and finance! 💯👌

2. Interest area identification.

We used a non-fixed-width font in this experiment because it better reflects the actual chatbots used in real life. This meant that letters had different widths in the experiment, which varied slightly depending on which letters were adjacent (‘kerning’).

 The heights of lines, text box borders, and the width of each unique letter and punctuation mark, plus their bigrams, as well as each emoji used in the experiment, was measured manually from a video of a conversation with the chatbot. In this conversation, each upper and lowercase letter, number, and four punctuation marks (which were ,.:-) were pasted into the participant text box separated by spaces, followed by the bigram pairings of each. Then, all of the emoji used in the experiment were pasted into the participant text box, as well as all of the numbers and other special characters used in the experiment (which were ?¿!¡%&~</()).

Still frames were extracted from this video, and then each element was measured in pixels using GIMP. Characters on their own (e.g. A) were used to provide a measurement called ‘char-as-left’ that reflected the width of the character when presented in isolation or as the first character in a word. All bigrams (e.g. AA, Aa, A;) were then measured. The ‘char-as-left’ measurement for each bigram was subtracted to leave the width of the right character; these were then averaged together across for each right character to create a measurement called ‘char-as-right’.  The punctuation characters measured only in unigram form were given a ‘char-as-right’ measurement taken from a width-matched letter, for simplicity. All numbers were then transformed to map from the video frame size to the original screen size, which differed because of the display settings required for WebLink.

The lookup table for each character is as follows:

​​Table 1. The lookup table

These values were used to map x and y coordinates within specific interest areas with the following R scripts:

lookup_table <- read.table('char_widths.txt', header=T,sep="\t",comment.char = "",quote="",encoding='UTF-8')

## tag which characters are emojis-- we'll treat them separately

## they're first in the table

## note that R doesn't handle the gendered emojis well-- it splits them into the NB version+gender sign.

## the gender signs are treated here therefore as a special zero-width emoji character.

emojis <- lookup_table[1:136,]

## this function turns some text string into parsed words+spaces and emoji, with defined w and h of each segment

## spaces between emoji will be defined as their own word

## input is string of text

input2wh <- function(txt_string){

## txt_string <- "I want🍕🍕"

txt_string_chars <- unlist(strsplit(txt_string, ""))

## separate into segments: words plus any following punctuation or spaces, and all emoji as separate

txt_emoji <- which(txt_string_chars %in% emojis[,1] == T)

txt_emoji <- c(txt_emoji,txt_emoji-1) ## add a boundary before any emoji in case one wasn't found

txt_spaces <- unlist(gregexpr(" ",txt_string))

## add in all the words, all the emojis, and the final character, remove any duplicates

txt_end <- unique(sort(c(txt_emoji,txt_spaces,length(txt_string_chars))))

## remove any zeroes which get inserted if the emoji is in first position

## or -1 which get inserted if a single word is entered

txt_end <- if(txt_end[1]==0){txt_end[2:length(txt_end)]}else{txt_end}

txt_end <- if(txt_end[1]==-1){txt_end[2:length(txt_end)]}else{txt_end}

txt_start <- unique(c(1,(txt_end + 1))) ## except for the first, words start after the ends we found; omit the last

## parse these units from the text

txt_words <- vector(length=(length(txt_end)))

for(k in 1:length(txt_words)){

  txt_words[k] <- substring(txt_string,txt_start[k],txt_end[k])

}

## then make a matrix that has each word and its width and height

out_string <- matrix(nrow=length(txt_words),ncol=3)

out_string[,1] <- txt_words

colnames(out_string) <- c('word','w','h')

for(j in 1:length(txt_words)){

  word_char <- unlist(strsplit(txt_words[j],""))

  for(i in 1:length(word_char)){

    if(i==1){

      wi = lookup_table[which(lookup_table$char == word_char[1]), ]$w.as.left

      hi = lookup_table[which(lookup_table$char == word_char[1]), ]$h

    }

    else{

      w2=lookup_table[which(lookup_table$char == word_char[i]), ]$w.as.right

      wi=wi+w2

      h2 = lookup_table[which(lookup_table$char == word_char[i]), ]$h

      hi = max(hi, h2)

    }

  }

  ## error handling for gendered emojis-- I'm having R treat them as the base and a zero-width additional element separated by a 'nothing'

  if(length(wi)==0){wi=0}else{}

  if(length(hi)==0){hi=0}else{}

  out_string[j,2]=wi

  out_string[j,3]=hi

}

out_string

}

input2wh("I like🥑💕")

## this function takes an input and converts it to xy coords in a textbox that might or might not wrap

## and might or might not be defined by center=0,0 (or top left = 0,0)

## inputs: a max text box width, space between lines, border at left, and top of textbox (in pixels)

## also the text string and whether it needs to be center-coordinate defined

wh2xy <- function(maxwidth,linespace,leftborder,topborder,txt_string,centercoord=T){

  #txt_string <- "I want 🍕🍕"

  #maxwidth <- 37

  #linespace <- 2

  #leftborder <- 10

  #topborder <- 10

  ## first, do all of the previous function:

  txt_string_chars <- unlist(strsplit(txt_string, ""))

  ## separate into segments: words plus any following punctuation or spaces, and all emoji as separate

  txt_emoji <- which(txt_string_chars %in% emojis[,1] == T)

  txt_emoji <- c(txt_emoji,txt_emoji-1) ## add a boundary before any emoji in case one wasn't found

  txt_spaces <- unlist(gregexpr(" ",txt_string))

  ## add in all the words, all the emojis, and the final character, remove any duplicates

  txt_end <- unique(sort(c(txt_emoji,txt_spaces,length(txt_string_chars))))

  ## remove any zeroes which get inserted if the emoji is in first position

  ## or -1 which get inserted if a single word is entered

  txt_end <- if(txt_end[1]==0){txt_end[2:length(txt_end)]}else{txt_end}

  txt_end <- if(txt_end[1]==-1){txt_end[2:length(txt_end)]}else{txt_end}

  txt_start <- unique(c(1,(txt_end + 1))) ## except for the first, words start after the ends we found; omit the last

  ## parse these units from the text

  txt_words <- vector(length=(length(txt_end)))

  for(k in 1:length(txt_words)){

    txt_words[k] <- substring(txt_string,txt_start[k],txt_end[k])

  }

  ## then make a matrix that has each word and its width and height

  out_string <- matrix(nrow=length(txt_words),ncol=3)

  out_string[,1] <- txt_words

  colnames(out_string) <- c('word','w','h')

  for(j in 1:length(txt_words)){

    word_char <- unlist(strsplit(txt_words[j],""))

    for(i in 1:length(word_char)){

      if(i==1){

        wi = lookup_table[which(lookup_table$char == word_char[1]), ]$w.as.left

        hi = lookup_table[which(lookup_table$char == word_char[1]), ]$h

      }

      else{

        w2=lookup_table[which(lookup_table$char == word_char[i]), ]$w.as.right.raw

        wi=wi+w2

        h2 = lookup_table[which(lookup_table$char == word_char[i]), ]$h

        hi = max(hi, h2)

      }

    }

    ## error handling for gendered emojis-- I'm having R treat them as the base and a zero-width additional element separated by a 'nothing'

    if(length(wi)==0){wi=0}else{}

    if(length(hi)==0){hi=0}else{}

    out_string[j,2]=wi

    out_string[j,3]=hi

  }

## now convert this to x and y coord pairs

## wrap lines if the xmax is bigger than the textbox

out_seq <- cbind(out_string[,1],rep(leftborder,dim(out_string)[1]),out_string[,2],rep(topborder,dim(out_string)[1]),as.numeric(as.character(out_string[,3]))+topborder,rep(1,dim(out_string)[1]))

colnames(out_seq) <- cbind('word','xmin','xmax','ymin','ymax','line')

for(m in 1:dim(out_string)[1]){

  if(m==1){  }else{

  ## establish xmin and xmax per word

  out_seq[m,2] <- out_seq[(m-1),3]

  out_seq[m,3] <- as.numeric(as.character(out_seq[m,3])) + as.numeric(as.character(out_seq[(m-1),3]))

  ## if xmax is too wide, start a new line

  ## this means adding previous ymax and line spacing to ymin, and subtracting to 0 for xmin

    if(as.numeric(as.character(out_seq[m,3])) > maxwidth){

    out_seq[m,6] <- as.numeric(as.character(out_seq[(m-1),6]))+1

    out_seq[m,4] <- as.numeric(as.character(out_seq[(m-1),5])) + linespace

    out_seq[m,5] <- as.numeric(as.character(out_seq[(m),4])) + as.numeric(as.character(out_string[m,3]))

    out_seq[m,2] <- leftborder

    out_seq[m,3] <- out_string[m,2]

    }else{

      out_seq[m,4:6] <- out_seq[(m-1),4:6] ## if we didn't correct the xmax/xmin, make sure the y and line values inherit from above

  }

}

}

if(centercoord==T){

    out_seq[,2:3] <- as.numeric(as.character(out_seq[,2:3])) - (maxwidth + 2*leftborder)/2

    out_seq[,4:5] <- as.numeric(as.character(out_seq[,4:5])) - (max(as.numeric(as.character(out_seq[,5]))) + topborder)/2

}else{

}

wnum <- 1:dim(out_seq)[1]

out_seq <- cbind(out_seq,wnum)

out_seq

}

3. Consent Form

Participant consent was obtained prior to the experiment. The form included information that the research involves the use of artificial intelligence and large language models, specifically ChatGPT. This included information about ChatGPT such as explanation of its limitations, and data privacy policies. The consent form mentioned that the participant’s responses will be passed to OpenAI to generate a response based on their input, and that they are free to provide to the chatbot as little detail as they feel comfortable with to maintain their anonymity. The IRB approved an exempt protocol №3-23-0229.

Consent form

We are asking you to participate in a research study titled “How Gen Z Users Engage with Chatbots: An Eye-Tracking Study”. This study is being led by Marina Zhukova and Dr. Laurel Brehm from the UCSB Department of Linguistics.

What the study is about. In this study, we will evaluate how Gen Z users process messages from a chatbot, using an eye-tracking software. Collecting this data will help us understand where people focus their attention when engaging with a chatbot and how people perceive the messages from the chatbot.

What we will ask you to do. In this study, you will be asked to engage in a 10 min conversation with a pre-programmed chatbot on a range of selected topics (e.g., lifestyle, hobbies, or technologies) using OneReach.ai chatbot interface. During the interaction with the chatbot, your eye movements will be tracked with an eye-tracking device (camera). Prior to the conversation with the chatbot, we will calibrate the eye-tracking device. After the conversation with the chatbot, we will ask you several questions (demographic and conversation-related). The estimated length of the study is 30 minutes.

Eligibility. You must be based in the Santa Barbara area, consider yourself a fluent speaker of English, and be 18-26 years old. You also need to be free of language disorders (e.g. dyslexia), and to have normal vision or vision corrected with contact lenses (NOT glasses).

Incentives for participation  For your participation, you will be given a $10 Amazon, Visa, or Target gift voucher. This will be emailed to you after completion of the study.

Privacy/Confidentiality/Data Security. During your conversation with the chatbot, chatbot responses will be generated by ChatGPT. Your responses will be passed to OpenAI to generate a response based on your input. You are free to provide as much or as little detail as you like in your responses to the chatbot in order to maintain your anonymity. If results of this study are published or presented, any personally identifiable information will not be used. The study is anonymous and does not collect any directly identifying information.

Taking part is voluntary. Your participation in this study is voluntary. You may refuse to participate before the study begins or discontinue at any time by exiting the interface prior to completing the study. If you choose to do this, there will be no penalty and no effect on the compensation earned before withdrawing, or your academic standing, record, or relationship with UCSB.

Risks and discomforts. We do not anticipate any risks beyond those of everyday life from participating in this research. The eye-tracker used in the study operates using infrared light and the total amount of infrared light that the participant will be exposed to is considered to be safe and is of no more risk to you than being outside on a sunny day.

ChatGPT Use and Limitations. ChatGPT, or the third-generation Generative Pre-trained Transformer, is a neural network machine learning model trained using internet data to generate any type of text. During your conversation with the chatbot, chatbot responses will be generated by ChatGPT. Your responses will be passed to OpenAI to generate a response based on your input. While ChatGPT generates coherent and grammatically correct sentences, it lacks common sense and real-world knowledge. This can sometimes result in nonsensical or incorrect outputs. Although ChatGPT generates text based on context, it can have limitations in understanding the full context of a situation.

If you have questions. If you have any questions or concerns about your rights and treatment as a research subject, please contact Marina Zhukova at mzhukova@ucsb.edu or Dr. Laurel Brehm at lbrehm@ucsb.edu.If you have any questions regarding your rights and participation as a research subject, please contact the Human Subjects Committee at +1 805 893-3807 or hsc@research.ucsb.edu. Or write to the University of California, Human Subjects Committee, Office of Research, Santa Barbara, CA 93106-2050.

Checking this box indicates that you have read the above information, willing to participate,meet the eligibility requirements for the study, and have received answers to any questions you asked. ____  Yes, I consent to take part in the study.

4. Tables (data analysis)

Table 2. Model coefficients: the impact of the POS, word length, question, sentiment, hallucination on the first fixation duration

Table 3. First Fixation Duration by POS: calculated mean, standard deviation,

and confidence intervals

Table 4. Model coefficients: the impact of the bot persona and of the individual emoji

on the total revisit duration

Table 5. Total Visit Duration by Chatbot-Specific Emoji: calculated mean, standard deviation, and confidence intervals

5. Figures

Figure 1. Total visit duration for emojis by chatbot. Millennial chatbot was associated with shorter total visit durations on emojis compared to the GenZ chatbot.